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ABSTRACT 

This  report  covers  the  effort  of  Data- Design  Laboratories  'inder  Contract 
AF  41(60S)-2743.  Project-Task  771906.  The  contract  covers  the  production  of 
1 00  mechanical-principles  test  items  and  50  background- for-world-events  test 
items.  Fourteen  50-item  tests  were  submitted  for  tryouts  to  groups  of  officer 
candidates  who  scored  above  the  65th  percentile  on  the  Armed  Forces  Qualifica¬ 
tion  Test.  The  average  group  size  was  383.  Items  were  scored  on  difficulty,  phi 
for  each  alternative,  and  phi  ratio  for  the  correct  alterr  ative. 

Considerable  difficulty  was  encountered  in  obtaining  the  required  number 
of  acceptable  questions.  It  is  believed  that  this  difficulty  stemmed  primarily 
from  the  lack  of  an  external  criteria  and.  to  a  lesser  extent,  from  the  statistical 
tftsts  specified. by  contract. 

These  difficulties  required  an  extension  of  contract  schedules  and  a  total  of 
700  items  for  development  of  150  acceptable  items. 
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I.  PREFACE 

This  is  the  final  report  describing  Data-Design  Laboratories :  activities  under  contract 
AF  41(609}-2743,  Project-Task  771906.  This  report  summarizes  all  activities  during  the 
period  15  December  1964  through  15  September  1966. 


H.  TASK  DESCRIPTION 

This  task  involved  the  preparation  and  evaluation  of  test  items  to  be  used  in  the  prepa¬ 
ration  of  officer  selection  and  classification  test  batteries.  A  total  of  150  fully  acceptable 
items  were  required:  100  mechanical-principles  items  and  50  background-for-world-events 
items.  Qualification  of  the  items  for  acceptability  was  as  follows: 

Correlation.  A  minimum  acceptable  correlation  of  the  correct  alternative  with  the  total 
test  score  to  be  shown  by  a  ratio  of  0.45  or  more  between  the  obtained  correlation  (phi  co¬ 
efficient)  and  the  maximum  possible  phi  coefficient  for  the  difficulty  of  the  item.  The  item 
statistics  were  to  use  a  phi  coefficient  based  upon  the  upper  and  lower  50%  of  the  sample  of 
examinees. 

Distribution  of  Difficulty  Level.  The  tabulation  below  show’s  the  required  distribution  of 
difficulty  level  veisus  percent  of  total  items  of  each  type.  Difficulty  level  was  computed  as 
percent  answering  an  item  correctly. 


Difficulty  Level 


70-55 

60-69 

50-59 

40-49 

30-39 

15-29 


Percent  of  Items 

3%  or  less 
8  to  12% 

25  to  30% 

30  to  35% 

20  to  25% 

5%  or  less 


m.  DESCRIPTION  OF  WORK  ACCOMPLISHED 


Following  agreement  with  the  Air  Force  on  style  and  content  of  the  sample  questions 
(Phase  I),  Data-Design  proceeded  with  development  of  items  (Phase  II)  for  tryouts  (Phase 
HI)  and  statistical  analysis  (Phase  IV).  The  following  paragraphs  present  details  of  this 
process,  including  preparation  of  the  item  statistics. 
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Test  Booklets.  A  total  of  14  test  booklets  were  prepared  during  the  course  of  this  task 
(Table  1).  Each  test  booklet  consisted  of  50  multiple-choice  questions,  or  items,  o:i  either 
background  for  world  events  or  mechanical  principles.  The  mechanical  principles  test  book¬ 
lets  involved  a  series  of  apprc  r  10  diagrams  of  devices  (mechanisms),  with  from  three 

to  six  questions  concerning  each  device.  From  three  to  five  alternatives  were  included  for 
each  item,  with  most  having  either  four  or  five  alternatives.  The  booklets  were  reproduced 
(400  copies  each)  and  forwarded  to  Lackland  AFB  for  tryouts. 

Tryouts.  The  test  booklets  were  administered  by  Air  Force  personnel  to  selected  groups 
of  examinees.  By  contract  these  examinees  ranked  in  or  above  the  65th  percentile  on  the 
Armed  Forces  Qualification  Test,  showing  above  average  mentality.  Answer  sheets  from 
the  tryouts  were  returned  to  Data-Design  Laboratories  for  scoring  and  item  analysis.  Sample 
size  for  each  test  booklet  is  shown  in  Table  1. 

Item  Analysis.  The  test  results  were  graded  and  rank-ordered  according  to  number  of 
correct  responses  on  each  answer  sheet.  The  median  score  was  located,  and  the  results 
divided  into  upper  and  lower  groups.  Responses  to  individual  items  were  then  tabulated, 
percent  difficulty  and  phi-coefficient-ratio  computed  for  each  item,  and  finally,  actual  phi 
coefficients  computed  for  the  alternatives  of  each  item  meeting  the  qualifications  listed  pre¬ 
viously. 


TABLE  1.  TEST  BOOKLETS 


Exam  No. 

Title 

No.  Subjects 

PL  6501  RC 

BWE  A 

361 

PL  6504  RC 

BWE  B 

386 

PL  6507  RC 

BWE  C 

379 

PL  6:  '  0  RC 

BWE  D 

396 

PL  6512  RC 

BWE  E 

397 

PL  6515  RC 

BWE  F 

391 

PL  6516  RC 

BWE  G 

369 

PL  6502  RC 

MP  A 

414 

PL  6505  RC 

MP  B 

395 

PL  6508  RC 

MP  C 

367 

PL  6511  RC 

MP  D 

395 

PL  6513  RC 

MP  E 

389 

PL  6517  RC 

MP  F 

369 

PL  6601  RC 

MP  G 

379 
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Results.  The  initial  results  were  disappointing.  As  noted  in  the  quarterly  progress 
reports,  the  first  few  test  booklets  yielded  very  small  numbers  of  acceptable  items,  and 
that  these  were  in  the  'Very  easy"  and  'Very  hard"  difficulty  ranges.  Subsequent  test 
booklets  continued  to  yield  fewer  acceptable  items,  particularly  in  the  mechanical  prin¬ 
ciples  category.  A  great  deal  of  effort  was  expended  in  examining  and  comparing  accept¬ 
able  versus  unacceptable  items.  In  no  case  was  there  any  discernible  difference  of  wording 
or  style  or  type  which  provided  helpful  direction.  During  a  visit  to  the  contract  Research 
Psychologist  (in  October  1965)  it  was  learned  that  a  one-for-three  or  one-for-four  ratio  of 
acceptable-to-supplied  items  was  common  in  his  experience. 

During  the  above  visit  a  number  of  methods  for  improving  the  yield  from  the  test 
booklets  were  discussed.  These  methods,  which  are  described  below,  proved  more  fruitful 
and  eventually  enabled  completion  of  the  task. 

Method  1.  The  first  method  involved  purification  of  the  test  through  elimination  of  the 
items  which  discriminated  very  poorly;  i.  e. ,  had  low-positive  or  negative  phi  coefficients. 
This  procedure  worked  very  well  for  the  background-to-world-events  tests. 

Methods  2  and  3.  These  methods  involved  increasing  homogeneity  of  subject  matter  and 
item  difficulties,  respectively.  Neither  method  was  actually  tried  with  the  BWE  tests,  due 
to  success  with  the  first  method.  Method  3,  increasing  homogeneity  of  the  item  difficulties, 
provided  little  improvement  in  mechanical  principles  items. 

Method  4.  This  method,  which  was  used  only  with  the  mechanical  principles  test  items, 
involved  viewing  the  test  booklet  as  containing  a  series  of  subtests,  each  concerning  a  single 
mechanical  device.  Method  4  was  successful  in  varying  degrees,  ranging  from  95  percent 
yield  down  to  abjut  30  percent  yield;  the  best  yield  was  obtained  when  the  difficulty  levels 
of  the  items  u^ed  for  a  given  device  were  fairly  homogenous. 

Final  Delivery.  The  acceptable  items  obtained  from  the  tryout  and  item  analysis  process 
above  were  typed  on  provided  item  submittal  cards,  each  containing,  in  addition  to  the  item, 
the  item  statistics  (phi  coefficients,  phi  ratio  for  the  correct  alternative,  and  item  difficulty) 
and  required  identifying  information  (test  origin,  scoring  criterion,  item  number,  description 
of  the  sample  used  during  tryou<s,  and  the  tryout  date).  Where  required,  alternative  distract- 
ors  were  typed  below  the  item  as  possible  replacements  for  the  few  distractors  which  corre¬ 
lated  positively  with  the  criterion.  The  related  artwork  for  the  mechanical  principles  items 
were  identified  on  the  item  cards  (MP-1,  MP-2,  etc).  Original  artwork,  a  negative,  and  two 
glossy  prints  of  each  device  were  included  in  the  final  submittal  package  in  individual  en¬ 
velopes;  each  envelope  was  marked  with  its  identification  number  (again,  MP-1,  MP-2,  etc). 
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